Black-box optimization has emerged as an effective technique in many real-world applications to find the global optimum of expensive, noisy black-box functions. Some notable applications include hyper-parameter optimization in machine learning algorithms Snoek et al. [2012], Bergstra and Bengio [2012], synthesis of short polymer fiber materials, alloy design, 3D bio-printing, and molecule design Greenhill et al. [2020], Shahriari et al. [2015], optimizing design parameters in computational fluid dynamics Morita et al. [2022], and scientific research (e.g., multilayer nanoparticle, photonic crystal topology) Kim et al. [2022]. Bayesian Optimization is a popular example of black-box optimization method. Typically, Bayesian Optimization algorithms use a probabilistic regression model, such as a Gaussian Process (GP), trained on existing function observations. This model is then utilized to create an acquisition function that balances exploration and exploitation to recommend the next evaluation point for the black-box functions. Various options exist for acquisition functions, including improvement-based methods like Probability of Improvement Kushner [1964], Expected Improvement Mockus et al. [1978], the Upper Confidence Bound Srinivas

With increasingly more hyperparameters involved in their training, machine learning systems demand a better understanding of hyperparameter tuning automation. This has raised interest in studies of provably black-box optimization, which is made more practical by better exploration mechanism implemented in algorithm design, managing the flux of both optimization and statistical errors. Prior efforts focus on delineating optimization errors, but this is deficient: black-box optimization algorithms can be inefficient without considering heterogeneity among reward samples. In this paper, we make the key delineation on the role of statistical uncertainty in black-box optimization, guiding a more efficient algorithm design. We introduce \textit{optimum-statistical collaboration}, a framework of managing the interaction between optimization error flux and statistical error flux evolving in the optimization process. Inspired by this framework, we propose the \texttt{VHCT} algorithms for objective functions with only local-smoothness assumptions. In theory, we prove our algorithm enjoys rate-optimal regret bounds; in experiments, we show the algorithm outperforms prior efforts in extensive settings.

Black-box optimization is essential for tuning complex machine learning algorithms which are easier to experiment with than to understand. In this paper, we show that a simple ensemble of black-box optimization algorithms can outperform any single one of them. However, searching for such an optimal ensemble requires a large number of experiments. We propose a Multi-GPU-optimized framework to accelerate a brute force search for the optimal ensemble of black-box optimization algorithms by running many experiments in parallel. The lightweight optimizations are performed by CPU while expensive model training and evaluations are assigned to GPUs. The multi-GPU solution achieves 10x speedup of the CPU implementation.